Learning Contextual Representations for Semantic Parsing with Generation-Augmented Pre-Training

Most recently, there has been significant interest in learning contextual representations for various NLP tasks, by leveraging large scale text corpora to train powerful language models with self-supervised objectives, such as Masked Language Model (MLM). Based on a pilot study, we observe three issues of existing general-purpose when they are applied the text-to-SQL semantic parsers: fail detect column mentions utterances, infer from cell values, and compose target SQL queries complex. To mitigate these issues, present model pretraining framework, Generation-Augmented Pre-training (GAP), that jointly learns natural utterance table schemas, generation generate high-quality pre-train data. GAP is trained 2 million utterance-schema pairs 30K utterance-schema-SQL triples, whose utterances generated models. experimental results, neural parsers leverage representation encoder obtain new state-of-the-art results both Spider Criteria-to-SQL benchmarks.